Laplacian smoothing gradient descent

We propose a class of very simple modifications gradient descent and stochastic leveraging Laplacian smoothing. show that when applied to large variety machine learning problems, ranging from logistic regression deep neural nets, the proposed surrogates can dramatically reduce variance, allow take larger step size, improve generalization accuracy. The methods only involve multiplying usual (stochastic) by inverse positive definitive matrix (which be computed efficiently FFT) with low condition number coming one-dimensional discrete or its high-order generalizations. Given any vector, e.g., smoothing preserves mean increases smallest component decreases largest component. Moreover, we optimization algorithms these converge uniformly in Sobolev $$H_\sigma ^p$$ sense optimality gap for convex problems. code is available at: https://github.com/BaoWangMath/LaplacianSmoothing-GradientDescent .